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Abstract 

This work extends a recently developed mathematical theory of thermodynamics for Markov 
processes with, and more importantly, without detailed balance. We show that the Legendre 
transform in connection to ensemble changes in Gibbs' statistical mechanics can be derived from 
the stochastic theory. We consider the joint probability pxY of two random variables X and Y 
and the conditional probability Px\Y=y*-, with y* = (Y) according to pxY- The stochastic free 
energies of the XY system (fluctuating Y ensemble) and the X\Y system (fixed Y ensemble) are 
related by the chain rule for relative entropy. In the thermodynamic limit, defined as V,Y ^ 00 
(where one assumes Y as an extensive quantity, while V denotes the system size parameter), 
the marginal probability obeys pviy) — )• exp(— y(/)(z)) with z = y/V. A conjugate variable /x = 
—<p{z) I z naturally emerges from this result. The stochastic free energies of the fluctuating and fixed 
ensembles are then related by Fxy = Fx\Y=y* — fJ'y*, with p, = dFx\Y=y/ dy. The time evolutions 
for the two free energies are the same: d[FxY{t)\/ dt = d[Fx\Y=y*(t){'t)\/ dt. This mathematical 
theory is applied to systems with fixed and fluctuating number of identical independent random 
variables (idea gas), as well as to microcanonical systems with uniform stationary probability. 
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I. INTRODUCTION 



Recently, a rather complete statistical thermodynamic structure for general Markov dy- 
namics has been revealed [l-4|. This formalism provides a novel perspective to investigate 
the dynamics of Markov processes in terms of two emergent quantities, entropy and free 
energy, which satisfy certain inequalities jsj. It has been shown that the entropy production 
of a Markov dynamics can be mathematically decomposed into two non-negative terms: 
free energy dissipation corresponding to Boltzmann's original idea of irreversibility of spon- 
taneous change, and house-keeping heat (the energy input necessary to keep a system's 
stationary state away from equilibrium), corresponding to Brussels school's notion of irre- 
versibility in nonequilibrium steady states (NESS) j6-9|. The stochastic free energy also 
has another important property: It is conserved in one-to-one invertible dynamics, e.g., 
diffeomorphism, but decreasing in either many-to-one endomorphism or random dynamics 
id . Ill, with uncertainties either in the past or in the future. This last result has been 

nn 

associated with the Second Law of Thermodynamics [12, 113I |. 

Studies on this thermodynamic structure with respect to multiscale dynamics have shown 
that with time scale separation in coarse-graining, the non-equilibrium stochastic free en ergy 
is invariant across different scales, while the entropy increases with finer scales H, [isj . 
Recall that the "partition function invariance" across scales in term of conditional free 
energy (potential of mean force) is an essential feature of equilibrium statistical mechanics 
161] . Note as well that the Bogoliubov inequality — which states that a true free energy is 
never greater than the free energy of a trial Hamiltonian — corresponds to the information 
Jensen's inequality for two random variables X and Y with probability distributions fx{x) 
and frix) [ij: (ln/x(X)) > (ln/y(X)). This also means that (ln(/x(X)//y(X))) > 0, 
the left-hand-side of which is known as relative entropy, or KuUback-Leibler divergence |ll|. 

One of the most important results of J.W. Gibbs' statistical thermodynamic theory, 
based on various ensembles with different environmental constraints, is the "equivalence" of 
ensembles as analytical devices for the thermodynamics of the same macroscopic system. For 
example, the canonical ensemble with fixed and V {N being the number of particles and V 
the volume of a system), the grand canonical ensemble with fixed /i (the chemical potential 
of a particle reservoir) and V, and the isobaric ensemble with fixed N and pressure over the 
system p, all give the same macroscopic thermodynamic laws H, 13], including the Maxwell 
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relations. Still, for different ensembles, the corresponding thermodynamic potentials, the 
"free energies" for the Second Law, are different: entropy for the microcanonical ensemble, 
Helmholtz free energy for the canonical ensemble, Gibbs free energy for the isobaric ensemble, 
etc. The different thermodynamic potentials are related via Legendre transforms. However, 
their change in an irreversible process is the same entropy production which underlies the 
Second Law. 

In the present work we expand upon this thermodynamic formalism for Markov processes 
to include the notion of ensemble change a la Gibbs. The fluctuations in various ensembles 



with different environmental constraints are certainly different 
on a mesoscopic level (e.g. in "nanothermodynamics" 
a certain role in the thermodynamics of small systems 
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20|. This indicates that. 



21]Vthe finite-size effect has to play 



23[. One long noted phenomenon 



associated with this effect is the "entropy-enthalpy compensation" among different ensembles 
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24j . Such compensation seems to play a fundamental role in systems with multiple- 
scale dynamics [l^. Taking this into account, the present paper aims at studying the relation 
between different forms of stochastic free energy associated with different stochastic systems, 
and the rule for their transformation based on two key notions: conditional probability and 
thermodynamic limit. In fact, one is able to mathematically derive the well-known Legendre 
transform 25| in Gibbs' theory. We also show that while the forms of the nonequilibrium 



stochastic free energy for different ensembles are different, their time evolutions are the 
same. Attempts had been made previously to show the equivalence of ensembles in NESS 
Markov processes from a statistical perspective Q, but to our knowledge this is the 
first time that it is done from the standpoint of irreversible Markovian thermodynamics. We 



also apply the general mathematical 



ensembles, e.g., grand canonical 
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result to the transformation of free energies of different 



291] and microcanonical Markov systems. 



II. STOCHASTIC FREE ENERGY AND CONDITIONAL PROBABILITY 

We consider a Markov system with joint discrete random variables X and Y with 
probability mass function pxri"^, N,t) = PrjX^ = m, = A^}. This can be very gen- 
eral. One particular example, however, could be a system consisting of an arbitrary num- 
ber of particles, or subsystems, each of which can be in M different states, numbered 
1, ■ --M. Every particle can jump among all the available states following a continuous- 
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time, Markovian stochastic process. Let Yt be the number of particles at time t in the 



system and Xt = (xl^\--- , xl^\ ■ ■ ■ xj:^^^ ] be the corresponding particle composition: 



Xt^^ + ■ ■ ■ + Xl^' + ■ ■ ■ + Xt'^''' = Yt. The time evolution of the system corresponds to that of 
an irreducible continuous-time Markov process for non-negative integer-valued random vari- 
ables {X,Y)t, with joint probability mass function p^y(m, A^, t) = Pr{X( = Th,Yt = N}. 
Without introducing ambiguity, we shall drop the vector notation from X and m. 

According to [1], the stochastic free energy of the "fluctuating ensemble" with fluctuating 
X and Y is: 

pxY{m,N, t) 



-(Af) 



flu. ens 



it) 



oo oo 



2^2^PxyKAr,t)ln 

Ar=0 m=0 



Y,PY{N,t)Y,Px{m,t\N) In 



Af=0 
oo 



m=0 



Px{m,t\N) 
p^-^{m\N) 



+ ^py(Ar,t)ln 



N=0 



PY{N,t) 



(1) 



where Pxyii^, is the unique stationary distribution of the Markov system 30|. Eq. ([T]) 
is well-known in probability and information theory as the "chain rule for relative entropy" 



m 



12| . The stochastic free energy for the "fixed ensemble" with fluctuating X but fixed 
Yt = N is: 

px{m,t\N) 



p'^ximlN) 



P fix. ens. 

{t\N) = Y,Px{m,t\N) In 

m=l 

We are interested in the relationship between Ffiu,ens.{t) with mean value 

A^*(t) 



oo oo 

^^Arp^y(m,Ar,t), 

Ar=Om=0 



(2) 



(3) 



and Ffix^ens.{t\N) with fixed A^ = N*{t). In Gibbs' ensemble theory, they correspond to an 
ensemble change; e.g. grand-canonical to canonical ensembles (by fixing number of particles), 
canonical to microcanomical ensembles (by fixing the energy), and isobaric to canonical (by 
fixing the volume). We shall adopt the terminology of grand canonical ensemble in the rest 
of the paper, with X's being number of particles. The mathematics are the same if the X's 
represent energies, volumes, etc., as long as Y is an extensive quantity. 

Thermodynamic limit. By Y being an extensive quantity, we mean that the system 
has a natural parameter V such that Y scales with V. In many applications, this parameter 
can be the size of a system in terms of total numbers, spatial volumes, etc. The thermody- 
namic limit is then defined through the system's size V and the random variable Y. Both 



V and Y are extensive quantities and they tend to infinity, while Y/V becomes an intensive 



quantity and remains finite. Assuming the large deviation principle, one can assert that in 
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the thermodynamic limit there exists a function t), z = N/V, such that (see 
and references therein, as well as 33|): 

PY{N,t) ^ e-^'f'^'''\ (4) 

It is important to emphasize at this point that we assume the existence of the thermodynamic 
limit, as well as the validity of the large deviation principle. The novelty of the present work 
is to show that Gibbs' Legendre transform is a consequence of the stochastic free energy and 
the conditional probability. 

There are two scenarios that require separated treatments: {{) V ^ 00 first with finite 
time t < T, for a given T; and [ii) t — 00 first with finite V. In the "finite-time sce- 
nario" {i), the stochastic dynamics is essentially "trapped" in a local basin of attraction 
depending upon its initial condition; there is a broken ergodicity since the mean time for 



transition into another basin of attraction is on the order of e"^ with a > 3J]. In {ii), 
the system is always ergodic and only the global minimum of (f){z, 00) is relavant for a large 
system. Furthermore, except at critical conditions of phase transition, the function (j){z, 00) 
has a unique global minimum, z, to which Vpy{Vz, 00) will converge with probability 1: 
Vpy{Vz, 00) — > 6{z — z). Therefore, we can assume without loss of generahty that, in both 
cases, (t){z,t) has a single global minimum z*{t) for any given t. This means that pY{N,t) 
has a single dominant maximum at N*{t) = Vz*{t). From these considerations, we expect 
the distribution pY{N,t) ^ for most values except in the neighbourhood of A^*. For 
example, a Poisson distribution, in the limit of large A^*, has Py{N, t) ^ for most A^ values, 
except for those in the range [A^* — 7v^iV*, A^* + 7V A^*], where 7 > is a fixed constant. 
Thererfore we have 

00 

J2PY{N,t)Ffi,,,ns.{t\N) = Ff,,,,^s{t\N*{t)). (5) 

This is known as the "maximum term method" in classical statistical mechanics 
larly, one also has that 

'pY{N,t) 



35]. Simi- 



Y,PY{N,t) In 



7V=0 



_ p'y'{N) 



-lnp-(Ar*(t)) (6) 
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in the limit of large A^*, provided that Py{N, t) and Py{N) can be approximated by Poisson 
distributions (see Appendix A). Therefore, we have from Eqs. ([5]) and ([6]) that 

oo 

P flu. ens. 



N=0 

oo 



N=0 



' pY{N,t) 



Ffi,,ens{t\N*{t)) - lnp'y'{N*{t)). 



(7) 



Noting that pp{N) = e ^'t'^^^ is a probability distribution function, where 0(z) stands for 
(f){z, oo), and denoting z* = N*{t)/V, then Eq. ([7]) becomes. 



where 



Fflu.ensit) = Ffi,,ens{t\N*) - N*fx{z*), 



^(z) = -(t){z)/z. 



(8) 



(9) 



The emergent quantity yU is considered to be the conjugate variable to A^. 

We now apply large deviation theory to show that equation ^ corresponds to a Legendre 
transform. Observe that the Ihs of Eq. (JHl) has no explicit dependence upon V . Hence, ^i{z*) 
should be chosen in such a way that 

d 



dV 



ix.ens 

{t\Vz) -Vzfi{z)] = 0. 



That is, 



Therefore, 



OF, 



fix. ens 



dN 



-fi{z) 



Fflu.ens.it) = Ff,^,,ns.{t\N*{t)) - N* (t) fl{z*) , 

d ^ 



(10) 



(11) 



(12) 



^i{z* 



dN 



fix. ens 



m*it))- 



Neglecting the time t, this equation corresponds to the Legendre transform in classic ther- 
modynamic theory. 

The sign of fi. As a large deviation rate function, (f){z) is non- negative and its global 
minimum is at z where d(f){z)/dz = (f){z) = 0. Thus, dfi{z)/dz = fi{z) = 0. If the random 
variables X's are non-negative, so is the random variable Y. Then z is positive and /i in 
Eq. ([9]) is a non-positive quantity. For instance, if we consider Fji^.ens and Ffiu.ens in Eq. 
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(fT2|) to respectively be the negative entropy and the Helmholtz free energy divided by the 
temperature, then A^* is the internal energy and fi is the negative reciprocal temperature 
in classical thermodynamics. The relation between this mathematical result and the Third 
Law of Thermodynamics remains to be elucidated. 

Time evolution of the free energy. Eq. (IT^ also gives 



^^Fflu.ens. if) 



dFfi,,,ns.{t\N*{t)) 

dt 



(13) 



N* 

We see that even though the two free energies are different for different ensembles, their 
time derivative is idential. This is the Brussels school's insight: "even though there are 
different thermodynamic potentials for the Second Law for different ensembles, there is a 
unique entropy production." 

Maxwell relations. With the introduction of ^{z) in Eq. fll2p . one can obtain several 
Maxwell relations. Assume that Ffi^.ensiA^) is also a function of a second thermodynamic 
variable ^: Ffix.ensit\N ,^). Then, one immediately gets: 



(14) 



where ip = {dFfix.ens/ dC)N is the thermodynamic conjugate of ^ in a fixed ensemble. For 
example, if F is the Helmholtz free energy and ^ is the temperature T, then —ip is 5, the 
entropy. Furthermore, we get from Eq. (fT^ the following relation for the partial molar 
entropy: 

'''' m . (15) 



III. ENTROPY: ITS CHAIN RULE AND TIME EVOLUTION 



Entropy also follows a chain rule with conditional probability |ll| : 

oo 

ix.ens. 

N=0 

oo 

- J2pY{N,t)lnpy{N,t), 

N=0 

with 

Sfix.ensXt\N) = -J2Pxim,t\N) lnpxim,t\N). 

m=0 
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A similar line of derivation as above gives: 

oo 
N=0 

Therefore, Sjiu.ens.it) and Sfi^,ens.{t\N*{t)) are related by 

Sflu.ens.it) = Sf,,,,ns.it\N*{t)) + Sy(t), (16) 

where 

oo 

SY{t) = -Y,Pyi^'t) lnpy(iV,t). 

N=0 

The first term in the rhs of Eq. flTBl) corresponds to the entropy associated to the distribution 
of the N* fixed particles among the M available states, while the second term corresponds 
to the entropy associated to variations in the total particle count. 

If the peak region of pY{N,t) can be approximated by a Poisson distribution, then one 



can obtain an explicit S'y(t) ~ | ln[27reA^*(t)] for large N* values 36|. In the Appendix B, 
it is shown that for some examples, Sfix.ens can be on the order of In A^*(t) as well. 
Let us now calculate the time derivative: 

dSfl^.ens _ fdSf,,.ens.{t\N*)\ ^ f OS f,,,ens.{t\N*)\ dN* (t) 



dt V 9t J N* \ Jt dt 

+ (17) 

The additional terms in the entropy change are interpreted as follows: For a fixed ensemble, 
A^* is simply a system constant and s = {dS jix.ens/ dN*)t is the entropy change of the system 
due to the fluctuating ensemble. In Gibbs' theory, s is called the partial molar entropy. The 
last term in (ITTj) is, of course, the entropy change associated with fluctuations in total 
particle count. 

Comparing Eq. ( fTTl) and Eq. ( fT3l) . we see that the extra terms in entropy change have 
to precisely compensate the term with extra terms in energe change so that no extra term 



appears in Eq. (JT3l) . This is the essential idea of the entropy-enthalpy compensation 19l. l20| 



IV. MICROCANONICAL, CANONICAL AND GRAND CANONICAL MARKOV 
SYSTEMS 

We now apply the above general mathematical results to several special cases. 
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Canonical and grand- canonical Markov systems. Consider two Markov systems, 
one with a fixed number of identical, independent particles and one with a variable number 
of particles due to "birth" and "death" processes. The latter, a grand-canonical Markov 



system (gcMS), was introduced in 28|, |29|]. As an ideal gas, a gcMS consists of identical, 
independent particles each with M possible states. The state of a gcMS is thus represented 
by {uk \l < k < M}. The total number of particles in the system, ^flj^ rik = N, fluctuates. 
There are two types of "kinetic cycles" in a gcMS. The first type, which also occurs in a 
canonical Markov system (cMS), involves state transitions of particles without changing the 
total number of particles N, as shown in Fig. lA. The second type, which only occurs in 
gcMS, involves adding and removing a particle, as shown in Fig. IB. 




nkQkh (ni + 



(• • • ,?7.fc - 1, • • • ,ni + !,■■■ ,nh,---) 



nfc - 1, • • • ,ni, 




(A) 



(• • • - 1, • • • ,ne,---) 



fk 




rik-nk 



{■■■ ,nk,--- ,ni,---) 



nkQki 




{ne + l)f 



(• • • ,nfc - 1, • • • ,ni + !,■■■) 



FIG. 1: In a grand-canonical Markov system (gcMS), there are two types of kinetic cycles: (A) state 
transitions of particles without changing the total number of particles; and (B) cycles involving 
addition and removal of particles. For gcMS with detailed balance, one has qkiqmlhk = QikQhiQkh 
and i^kQkiVe = Vklik^i for each cycle type, respectively. 



We shall limit our discussions only to non-driven systems. The detailed balance conditions 
are qkiqihQhk = qekqheqkh, for the cycle in Fig. lA, and utqkeVe = i^eqekVk, for the cycle in 
Fig. IB. With detailed balance, the stationary distribution for the Markov system is easy 
to obtain. The first cycle type dictates a stationary distribution {p*\l < i < M} for a 
single particle independet of A^, such that p*qij = V^lji- Then the conditional probability 
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distribution is given by 

/"(m, . . . ,nM|iV) = ^' (pI)'^^ ■ ■ ■ (p;,)"- . (18) 

ni! ■ ■ ■ hm^- 

The detailed balance for the second type of cycles yields I'k / ijIkVV) — ^i/iViPe) = -^j which 
is a constant, and 

p^'^im, . . . , nfc, ■ ■ ■ , nM|iV)P^'(iV)nfcr7fe = p'^m, ■ ■ ■ , - 1, ■ ■ ■ , nM|iV - l)P^'(iV - l)z/fc. 
That is, 



p'§{N-l) rjkNpl N- 
This yields a Poisson distribution with mean A, the equilibrium mean particle count in the 

system. The same result obtained from the grand-canonical ensemble theory with In A being 

the chemical potential. Thus, the equilibrium distribution for fluctuating is 



/V„---,nM)=e-^n^^- (20) 

1=1 ^ 

From Eq. ( IT8|) . one has the following nonequilibrium stochastic free energy for an arbi- 
trary distribution p{{nk}\N) with fixed in a canonical Markov system (cMS) 1|: 



^ \p^<}{ni,--- ,nM N)J 

m,--- ,nM\N 

M , 

5^K)(-lnp*)-5^+ P(K}|A^)lnf 

k=\ ni,--- ,nM\N ^ 



in which the first term is the mean internal energy, the second term is the entropy S'at = 
— 'Y^p{{nk}\N)\n.p{{nk}\N), and the third term is the "entropy of mixing". 

From Eq. f l2U]) one finds that the non-equilibrium stochastic free energy for the gcMS 
with fluctuating A^, mean N*{t), and equilibrium mean A, is: 

~ ' rill ■ ■ ■ riAfl \ , , 

^ ^ (22) 



^gcA/5(i) = -X^('^fc)lnPfc - 5+ ^ p({r2fc})lnf 

fc=l i"-M 



e-A_\ni + ...+nM 



In the large number limit, the difference between S^^ and S = —^p{{nk})^'^p{{nk}) is 
of the order of (ln(A^))/2, which can be neglected. Regarding the third terms in Eqs. 
(I2T1) and ( l22l) . we show in Appendix C that in the thermodynamic limit, they differ by 
A^*(t)(lnA^*(t) - In A) - {N*{t) - A). This is precisely the Vcpiz) expected from Eq. ([HD, 
with (f){z) = zhi{z/z) — {z — z), z = N*{t)/V, and z = X/V (see also Appendix D). 
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In analogy with classical statistical mechanics, Fcms and FgcMS respectively correspond to 
the Helmholtz free energy and to negative pV. Therefore, their difference is a generalization 
of Gibbs' free energy: 

G{t) = Fg,Ms{t)-F,Ms{t\N*{t)) 

= N*{t) In - iN*{t) - A). (23) 

It is not hard to prove from ( 123|) that G > and that G = if and only if A^* = A. Moreover, 

dG dG dN*(t) dN* , N*(t) 

— = — = X In —. (24) 

dt dN* dt dt X ^ ^ 

Thus, we can see G{N) as a potential and dG/dN = ln(A^/A) as the corresponding chemical 
force, whereas the rate of change of G is given as the product of the force and the corre- 
sponding chemical flux dN/dt. The — A^* term in (1251) has repeatedly appeared in chemical 
kinetic literature for systems without conservation of total particle numbers js?]. 

Eq. ( 12^ therefore provide a justification for the thermodynamic formalism based on the 
theory of stochastic free energy. 

Finally, recall that FcMs{t) measures the Kullback-Leibler divergence between the cur- 
rent probability distribution p{{nk}\N* (t)) , and the corresponding stationary probability 
distribution, p'^'^{{nis}\N*{t)). Now, if we assume a separation of time scales such that the 
distribution p{{nk}\N*{t)) rapidly converges to p'^'^{{nk}\N*{t)), as compared with p{N,t), 
then FgcMsif) ~ and 

G{t) = Fg,Ms{t). 

Canonical and microcanonical Markov systems. We now consider a Markov 
system with joint stationary distribution for X and Y, with stationary Pxri^^y) — 
Px\Y=y(-^) (y) ■ further assume that the conditional probability Px\Y=yi^) ^^i" 

form, i.e. a microcanonical ensemble: 

Px\Y=yi^) = 7^- (25) 

Then, by applying the general formalism in Sec. HIl we have 

FxYit) = Y,Py{y)Fx\Y=y{t) + 5^ Py(y) In , (26) 

y y \ YiV)/ 

with 

Fx\Y=y = In fij; + ^ PX\Y=yix) \npx\Y=y{x) = Inily - Sx\Y=y (27) 
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Furthermore, in the thermodynamic hmit: 

where y*{t) = ^yVPyiy)- Therefore, 

Fxvit) = -Sx\Y=rit) + lnny4t) - \nPp'iy*it)) = -Sx\Y=rit) + Eiy*it)). (29) 

The steady-state "internal energy" E{y) thus satisfies Pyiy) = ^yC^'^^^^ which one can 
clearly recognize from the canonical ensemble theory in classical statistical mechanics (/crT 



1) — these results are in agreement with a theorem by van Campenhout and Cover [38 
In conclusion, we can see that the stochastic free energy approach provides yet another 
interpretation for the mathematics, complementary to that of Boltzmann or E.T. Jaynes 



39| 



V. SUMMARY 



In summary, in the thermodynamic limit with an infinite system size, the chain rule for 
relative entropies yields a Legendre transform for the stochastic free energy: Ffiu,ens(t) = 
Ffix.ensit\y*)—^^ Pyiy*)- In this relation, InPyiy) = —V(f){y/V) = N^, with y representing 
the system size, while n = —(y/y)(f){y/V) is the thermodynamic conjugate variable for the 
random variable Y. 

The time evolutions {d/dt)Ffiu,ens(t) = {d/dt)Ffixensit\y*{'^))- However, for the entropy 
we have that {d/dt)Sfiu.ens{t) = {d/dt)Sfix,ens{t\y*{t)) + s{t){d/ dt)y*{t) + {d/dt)SY, where 
s(t) = {d / dy*) S fix:,ens(t\y* (t)) IS the partial molar entropy. 

The general theory of nonequilibrium stochastic free energy and ensemble change, as- 
sociated with the conditional probability, can be applied to systems of fixed and variable 
numbers of identical random variables. For instance, the canonical and grand canonical sys- 
tems. In this case we obtain Ffi^,ens{t) = Fji^^ensit) + N*{t) (In N*{t) -In A) - {N*{t) - A), A 
being the equilibrium mean particle count of the system. The difference between Ffiu,ens{t) 
and Ffix,ens{t)i which respectively correspond to the —pV and the Helmholtz free energy in 
classical thermodjTiamics, can then be interpreted as the Gibbs free energy: the "chemical" 
work performed upon the system to change its particle count. 
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When we consider a Markov system with joint stationary X and Y, and a uniform con- 
ditional distribution Px\Y=y ~ ^y^^ obtain Fxrit) = —Sx\Y=y*it) + E{y*){t), where 
Sx\Y=y* is the entropy of the microcanonical system, and the "internal energy" satisfies 
Pyiu) = i^ye~^^y\ This example suggests a rather novel interpretation for Gibbs' canonical 
ensemble theory. From a stochastic Markov dynamics perspective, entropy and free energy 
are inherent concepts associated with a system's dynamics while the internal energy is 
an emergent concept for a canonical ensemble with a underlying microcanonical structure: 
E[y) = Inlfly/ Pp{y)) where Qy is Boltzmann's thermodynamic probability. This interpre- 
tation of the internal energy as an emergent concept is in sharp contrast to the logic usually 
presented in textbooks where it is given a priori based upon Newtonian mechanics. This 



view, however, is in agreement with 13|, in which a thermodynamic formalism based on 
free energy for "Darwinian dynamics" has been proposed. It was suggested that the canonic 
ensemble, i.e. Boltzmann- Gibbs type of distribution, is a natural consequence of Darwinian 
dynamics. 

The internal energy defined as above, E{y), is a function of the random variable. It is a 
fluctuating quantity. Its physical origin can lie outside the system, due to the interactions 
between the system and its environment. This should not be a surprise since in classical 
mechanics, potential energy can also be due to external forces, e.g. gravitation. 
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Appendix A: KuUback-Leibler divergence of Poisson distributions 

A Poisson distribution has the probabihty mass function (pmf): 

P,(A;) = -exp(-A), (Al) 
were A is the corresponding mean value. It follows that 

On the other hand, under the assumption that Ai ^ 1 and Stirling's formula ln(Ai!) ~ 
Al In Al — Al, we have 

ln(PA.(Ai)) ~ -Ailn (^^^ + (Ai - A2). (A3) 

We therefore obtain that 

E^A.(A;)ln(^) ^-ln(P,,(Ai)). (A4) 

Appendix B: Entropy for a system of N particles distributed in M different states 

We give an example that Sfi^.ens is on the order of InN. Again, considering total N 
idential, independent particles, each with M possible states. Let be the occupation 
numbers for the state i, X^i^i — ^- It is straightforward to demonstrate by induction 
that, given N and M, the number of possible, different occupation-number sets {rrii} is 
given by 

Ar!(M-l)! ■ ^ ^ 

The maximum possible entropy occurs when every occupation-number set has the same 
probability. Therefore, it is given by 

^-"^ ^ ^^Kiif- ijr ^^^^ 

In the limit when > M, {N + M - 1)\/N\ c:^ N^-'^, and so 

Smax ~ (M - 1) In AT - ln(M - 1)! ~ (M - 1) InN. (B3) 
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Appendix C: Entropy of mixing for the grand canonical and canonical Markov 
systems 



Consider the term 



r,cM5= E P(K})l"( g-A!^ni+!+nM ) 



ni,--- ,nM 



By making use of the conditional probabihty it can be rewritten as 

r.cM5 = EMA^) E p(K}|iV)ln( ^^'-^7^' ) -^p(7V)ln(^) . (CI) 

N ni,-,nM\N ^ ' / iV \ " / 

Noting that the Poisson distribution Px{N) satisfies 

-ln(P,(7V)) = - In (^^) ^ iVln (^^) - {N - X), 
when N ^ 1, Tgcus can be approximated by 

TgcMS - TcMS + N* In (^^^ + {N* - A), (C2) 
with N* = Arp(Ar) and 

r.M.= i; P(WliV)in(!^il^). 

ni,-,nM|JV* ^ ^ 

Appendix D: Asymptotic approximation of a Poisson distribution for large N 

For the Poisson distribution 

when N ^ 1, one can approximate 

A^e-^ 



(C3) 



Therefore, 



-ln(P;,(Ar)):^iVln(^) -(iV-A), (Dl) 



or, 



0(^) = -^]n{Px{N)) c^zln(^) -z + const., (D2) 
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in which z = N/V is the number density and z = X/V is the mean number density. Eq. 
(ID1|) is vahd for large with both large and small A. The form given in flD2|) has been 
known as the macroscopic free energy functional for a closed chemical reaction systems with 



mass-action kinetics 



371]. This is consistent with the fact that for a closed chemical kinetic 



system a multi-Poisson representation exists for its equilibrium distribution according to 
the Delbriick- Gillespie theory 34l |. 
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